{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Isolation Forest (IF) outlier detector deployment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wrap a scikit-learn Isolation Forest python model for use as a prediction microservice in seldon-core and deploy on seldon-core running on minikube or a Kubernetes cluster using GCP."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dependencies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- [helm](https://github.com/helm/helm)\n",
    "- [minikube](https://github.com/kubernetes/minikube) \n",
    "- [s2i](https://github.com/openshift/source-to-image) >= 1.1.13 \n",
    "\n",
    "python packages:\n",
    "- scikit-learn: pip install scikit-learn --> 0.20.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The outlier detector needs to detect computer network intrusions using TCP dump data for a local-area network (LAN) simulating a typical U.S. Air Force LAN. A connection is a sequence of TCP packets starting and ending at some well defined times, between which data flows to and from a source IP address to a target IP address under some well defined protocol. Each connection is labeled as either normal, or as an attack. \n",
    "\n",
    "There are 4 types of attacks in the dataset:\n",
    "- DOS: denial-of-service, e.g. syn flood;\n",
    "- R2L: unauthorized access from a remote machine, e.g. guessing password;\n",
    "- U2R:  unauthorized access to local superuser (root) privileges;\n",
    "- probing: surveillance and other probing, e.g., port scanning.\n",
    "    \n",
    "The dataset contains about 5 million connection records.\n",
    "\n",
    "There are 3 types of features:\n",
    "- basic features of individual connections, e.g. duration of connection\n",
    "- content features within a connection, e.g. number of failed log in attempts\n",
    "- traffic features within a 2 second window, e.g. number of connections to the same host as the current connection\n",
    "\n",
    "The outlier detector is only using 40 out of 41 features."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train locally\n",
    "\n",
    "Train on small dataset where you roughly know the fraction of outliers, defined by the \"contamination\" parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define columns to keep\n",
    "cols=['duration','protocol_type','flag','src_bytes','dst_bytes','land',\n",
    "      'wrong_fragment','urgent','hot','num_failed_logins','logged_in',\n",
    "      'num_compromised','root_shell','su_attempted','num_root','num_file_creations',\n",
    "      'num_shells','num_access_files','num_outbound_cmds','is_host_login',\n",
    "      'is_guest_login','count','srv_count','serror_rate','srv_serror_rate',\n",
    "      'rerror_rate','srv_rerror_rate','same_srv_rate','diff_srv_rate',\n",
    "      'srv_diff_host_rate','dst_host_count','dst_host_srv_count','dst_host_same_srv_rate',\n",
    "      'dst_host_diff_srv_rate','dst_host_same_src_port_rate','dst_host_srv_diff_host_rate',\n",
    "      'dst_host_serror_rate','dst_host_srv_serror_rate','dst_host_rerror_rate',\n",
    "      'dst_host_srv_rerror_rate','target']\n",
    "cols_str = str(cols)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python train.py \\\n",
    "--dataset 'kddcup99' \\\n",
    "--samples 50000 \\\n",
    "--keep_cols \"$cols_str\" \\\n",
    "--contamination .1 \\\n",
    "--n_estimators 100 \\\n",
    "--max_samples .8 \\\n",
    "--max_features 1. \\\n",
    "--save_path './models/'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test using Kubernetes cluster on GCP or Minikube"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the outlier detector as a model or a transformer. If you want to run the anomaly detector as a transformer, change the SERVICE_TYPE variable from MODEL to TRANSFORMER [here](./.s2i/environment), set MODEL = False and change ```OutlierIsolationForest.py``` to:\n",
    "\n",
    "```python\n",
    "from CoreIsolationForest import CoreIsolationForest\n",
    "\n",
    "class OutlierIsolationForest(CoreIsolationForest):\n",
    "    \"\"\" Outlier detection using Isolation Forests.\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "        threshold (float) : anomaly score threshold; scores below threshold are outliers\n",
    "    \"\"\"\n",
    "    def __init__(self,threshold=0.,load_path='./models/'):\n",
    "\n",
    "        super().__init__(threshold=threshold, load_path=load_path)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "MODEL = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pick Kubernetes cluster on GCP or Minikube."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "MINIKUBE = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if MINIKUBE:\n",
    "    !minikube start --memory 4096\n",
    "else:\n",
    "    !gcloud container clusters get-credentials standard-cluster-1 --zone europe-west1-b --project seldon-demos"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a cluster-wide cluster-admin role assigned to a service account named “default” in the namespace “kube-system”."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl create clusterrolebinding kube-system-cluster-admin --clusterrole=cluster-admin \\\n",
    "--serviceaccount=kube-system:default"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl create namespace seldon"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Add current context details to the configuration file in the seldon namespace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl config set-context $(kubectl config current-context) --namespace=seldon"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create tiller service account and give it a cluster-wide cluster-admin role."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl -n kube-system create sa tiller\n",
    "!kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller\n",
    "!helm init --service-account tiller"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check deployment rollout status and deploy seldon/spartakus helm charts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl rollout status deploy/tiller-deploy -n kube-system"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!helm install ../../../helm-charts/seldon-core-operator --name seldon-core --set usage_metrics.enabled=true   --namespace seldon-system"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check deployment rollout status for seldon core."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!kubectl rollout status deploy/seldon-controller-manager -n seldon-system"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Install Ambassador API gateway"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!helm install stable/ambassador --name ambassador --set crds.keep=false"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl rollout status deployment.apps/ambassador"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If Minikube used: create docker image for outlier detector inside Minikube using s2i. Besides the transformer image and the demo specific model image, the general model image for the Isolation Forest outlier detector is also available from Docker Hub as ***seldonio/outlier-if-model:0.1***."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "if MINIKUBE & MODEL:\n",
    "    !eval $(minikube docker-env) && \\\n",
    "    s2i build . seldonio/seldon-core-s2i-python3:0.4 seldonio/outlier-if-model-demo:0.1\n",
    "elif MINIKUBE:\n",
    "    !eval $(minikube docker-env) && \\\n",
    "    s2i build . seldonio/seldon-core-s2i-python3:0.4 seldonio/outlier-if-transformer:0.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Install outlier detector helm charts either as a model or transformer and set *threshold* hyperparameter value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if MODEL:\n",
    "    !helm install ../../../helm-charts/seldon-od-model \\\n",
    "        --name outlier-detector \\\n",
    "        --namespace=seldon \\\n",
    "        --set model.type=isolationforest \\\n",
    "        --set model.isolationforest.image.name=seldonio/outlier-if-model-demo:0.1 \\\n",
    "        --set model.isolationforest.threshold=0 \\\n",
    "        --set oauth.key=oauth-key \\\n",
    "        --set oauth.secret=oauth-secret \\\n",
    "        --set replicas=1\n",
    "else:\n",
    "    !helm install ../../../helm-charts/seldon-od-transformer \\\n",
    "        --name outlier-detector \\\n",
    "        --namespace=seldon \\\n",
    "        --set outlierDetection.enabled=true \\\n",
    "        --set outlierDetection.name=outlier-if \\\n",
    "        --set outlierDetection.type=isolationforest \\\n",
    "        --set outlierDetection.isolationforest.image.name=seldonio/outlier-if-transformer:0.1 \\\n",
    "        --set outlierDetection.isolationforest.threshold=0 \\\n",
    "        --set oauth.key=oauth-key \\\n",
    "        --set oauth.secret=oauth-secret \\\n",
    "        --set model.image.name=seldonio/mock_classifier:1.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Port forward Ambassador\n",
    "\n",
    "Run command in terminal:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "kubectl port-forward $(kubectl get pods -n seldon -l app.kubernetes.io/name=ambassador -o jsonpath='{.items[0].metadata.name}') -n seldon 8003:8080\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import rest requests, load data and test requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from utils import get_payload, rest_request_ambassador, send_feedback_rest, get_kdd_data, generate_batch\n",
    "\n",
    "data = get_kdd_data(keep_cols=cols,percent10=True) # load dataset\n",
    "print(data.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generate a random batch from the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "samples = 1\n",
    "fraction_outlier = 0.\n",
    "X, labels = generate_batch(data,samples,fraction_outlier)\n",
    "print(X.shape)\n",
    "print(labels.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Test the rest requests with the generated data. It is important that the order of requests is respected. First we make predictions, then we get the \"true\" labels back using the feedback request. If we do not respect the order and eg keep making predictions without getting the feedback for each prediction, there will be a mismatch between the predicted and \"true\" labels. This will result in errors in the produced metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "request = get_payload(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "response = rest_request_ambassador(\"outlier-detector\",\"seldon\",request,endpoint=\"localhost:8003\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the outlier detector is used as a transformer, the output of the anomaly detection is added as part of the metadata. If it is used as a model, we send model feedback to retrieve custom performance metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if MODEL:\n",
    "    send_feedback_rest(\"outlier-detector\",\"seldon\",request,response,0,labels,endpoint=\"localhost:8003\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Analytics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Install the helm charts for prometheus and the grafana dashboard"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!helm install ../../../helm-charts/seldon-core-analytics --name seldon-core-analytics \\\n",
    "    --set grafana_prom_admin_password=password \\\n",
    "    --set persistence.enabled=false \\\n",
    "    --namespace seldon"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Port forward Grafana dashboard"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run command in terminal:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "kubectl port-forward $(kubectl get pods -n seldon -l app=grafana-prom-server -o jsonpath='{.items[0].metadata.name}') -n seldon 3000:3000\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can then view an analytics dashboard inside the cluster at http://localhost:3000/dashboard/db/prediction-analytics?refresh=5s&orgId=1. Your IP address may be different. get it via minikube ip. Login with:\n",
    "\n",
    "Username : admin\n",
    "\n",
    "password : password (as set when starting seldon-core-analytics above)\n",
    "\n",
    "Import the outlier-detector-if dashboard from ../../../helm-charts/seldon-core-analytics/files/grafana/configs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run simulation\n",
    "\n",
    "- Sample random network intrusion data with a certain outlier probability.\n",
    "- Get payload for the observation.\n",
    "- Make a prediction.\n",
    "- Send the \"true\" label with the feedback if the detector is run as a model.\n",
    "\n",
    "It is important that the prediction-feedback order is maintained. Otherwise there will be a mismatch between the predicted and \"true\" labels.\n",
    "\n",
    "View the progress on the grafana \"Outlier Detection\" dashboard. Most metrics need the outlier detector to be run as a model since they need model feedback."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "n_requests = 100\n",
    "samples = 1\n",
    "for i in range(n_requests):\n",
    "    fraction_outlier = .1\n",
    "    X, labels = generate_batch(data,samples,fraction_outlier)\n",
    "    request = get_payload(X)\n",
    "    response = rest_request_ambassador(\"outlier-detector\",\"seldon\",request,endpoint=\"localhost:8003\")\n",
    "    if MODEL:\n",
    "        send_feedback_rest(\"outlier-detector\",\"seldon\",request,response,0,labels,endpoint=\"localhost:8003\")\n",
    "    time.sleep(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if MINIKUBE:\n",
    "    !minikube delete"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
